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Abstract 

Many queries that constrain multiple dimensions simultaneously are difficult to 
express and compute efficiently in both Structured Query Language (SQL) and multi- 
dimensional languages. We introduce the diamond cube operator, filling a gap among 
existing data warehouse operations. It aids decision support and analysis by facilitat- 
ing the execution of simultaneous multidimensional queries on data cubes. An example 
query from the business analysis domain follows. 

A company wants to close shops and terminate product lines simul- 
taneously. The CEO wants the maximal set of products and shops such 
that each shop would have sales greater than $400000, and each prod- 
uct would bring revenues of at least $100000 — assuming we terminate all 
other shops and product lines. 

Diamonds are intrinsically multidimensional and because of the interaction between 
dimensions the computation of diamond cubes is challenging. We present diamond 
dicing experiments on large data sets of more than 100 million facts. Our custom 
Java implementation is more than one hundred times faster, on a large data set, than a 
popular database engine. 

Keywords: OLAP, information retrieval, multidimensional queries 



1. Introduction 

Dealing with large data is challenging. One way to meet this challenge is to choose 
a sub-sample — select a subset of the data — with desirable properties such as repre- 
sentativity, conciseness, or homogeneity. In signal and image processing, software 
sub-samples data [ 1 1 for visualisation, compression, or analysis purposes: commonly, 
images are cropped to focus the attention on a particular segment. In databases, re- 
searchers have proposed similar sub-sampling techniques |f2j 0, including iceberg 
queries and top-A: queries QUI- 

Such reduced representations are sometimes of critical importance to get good on- 
line performance in Business Intelligence (BI) applications [4|. Even when perfor- 
mance is not an issue, browsing and visualising the data frequently benefit from re- 
duced views (9). 
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Figure 1: Sales (in million dollars) with a 5,10 sum-diamond shaded: stores need to 
have sales above $10 million whereas product lines need sales above $5 million. 

The computation of icebergs, top-fc elements, or heavy-hitters has received much 
attention |[T0l{l2"l . This type of query can be generalised so that interactions between 
dimensions are allowed. We propose a model that has acceptable computational costs 
and a theoretical foundation. We illustrate the proposal with examples from BI and 
from bibliometrics. 

BI example. In the BI context, consider Fig. [T] which represents the sales of different 
items in different locations. Typical iceberg queries might be requests for stores having 
sales of at least 10 million dollars or product lines with sales of at least 5 million dollars. 
However, what if the analyst wants to apply both thresholds simultaneously? He or she 
might contemplate closing both some stores and some product lines. In our example, 
applying the constraint on stores would close Chicago, whereas applying the constraint 
on product lines would not terminate any product line. However, once the shop in 
Chicago is closed, we see that the product line TV must be terminated which causes 
the closure of the Berlin store and the termination of two new product lines (Game 
console and DVD player). 

This multidimensional pruning query selects a subset of attribute values from each 
dimension that are simultaneously important. For all the attribute values remaining, 
every column (slice) in the location dimension sums to at least 10 and every row (slice) 
in the product dimension sums to at least 5. 

For ease of visualisation, the example above is provided in two dimensions, but the 
diamond cube operator is defined over d dimensions. For example, a three-dimensional 
query applied in the same BI context might constrain products, stores and manufactur- 
ers. 

The proposed operation is a diamond dice lfl3l 14 It produces a diamond, as 
formally defined in Section|3] Fig.[T]shows a "5,10 sum" diamond. 

Bibliometrics example. As another motivating example, consider a bibliographic table 
with columns for author, venue, and (first) keyword. Perhaps we want to analyze the 
publication habits of professors, but much work would be required to precisely identify 
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Table 1: Characteristics of selected diamonds from DBLP 
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which authors are professors. However, perhaps we can assume that most authors with- 
out at least 5 publications are not professors. A "5,1,1 count" diamond will exclude 
them. Moreover, suppose we worry that there might be venues that mostly cater to 
students. Publications in such venues should be excluded from analysis. The 5,50,1 di- 
amond will ensure that every retained person has published at least 5 articles in retained 
venues, and that every retained venue has at least 50 (presumed) professors publishing 
there. Not only will the 5,50,1 diamond prune some venues, but it may also remove 
some pseudo-professors who relied on student venues to achieve 5 publications. The 
5,50,10 diamond will additionally exclude publications on "fringe topics" — a query for 
that diamond is an attempt to identify publications on mainstream topics by professors 
in non-student venues. 

For illustration, we processed conference publication data available from DBLP fl5l : 
the data and details of its preparation are given elsewhere Ifl6l . See Table [1] for some 
characteristics of diamonds on this data. 

We can obtain other information about clusters of authors, venues and topics by 
requiring very large thresholds. For instance, the 119,119,1 diamond contains authors 
with 1 19 publications each in retained venues, and retained venues each have 1 19 pub- 
lications from retained authors. This diamond contains 1 1 prolific authors in the area of 
digital hardware and computer-aided design, who publish in 7 venues using 482 key- 
words. There are more authors (16) and venues (11) in the 100,100,2 diamond, but it 
is still very much a hardware area with 333 keywords. 

An extreme case is the 108,20,6 diamond, which involves the publications of au- 
thors I. Pomeranz and S. M. Reddy in 8 hardware venues. Within this diamond, the 
most frequent keyword is 'synchronous', which occurred 7 times more often than the 
least frequent, 'sequential'. Globally (i.e., in the 0,0,0 diamond), these keywords are 
almost equally frequent and are ranked 286th and 289th. These two authors are ranked 
20th and 11th, globally. 

Setting high thresholds is particularly useful in obtaining smaller, more easily an- 
alyzed, sets of data. This motivates the problem of finding a k,k,l (or k,k,2) diamond 
where the system is responsible for discovering that k = 119 (respectively, k = 100) 
results in the non-empty diamond with the fewest attribute values. 
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2. Related Work 



Diamonds are related to other work in various ways. They extract important sub- 
samples of the data as in Formal Concept Analysis or skyline and top-fc queries. Di- 
amonds generalise a two-dimensional iterative pruning algorithm [17] that trawls the 
Web for cyber-communities. We explore these relationships in the following sections. 

2.1. Trawling the Web for Cyber-communities 

A specialisation of the diamond cube is found in Kumar et al.'s work searching for 
emerging social networks on the Web ifTTl . Our approach is a generalisation of their 
two-dimensional ITERATIVE PRUNING algorithm. Diamonds are inherently multidi- 
mensional. Kumar et al. IfTTl model the Web as a directed graph and seek large dense 
bipartite subgraphs. A bipartite graph is dense if most of the vertices in the two dis- 
joint sets, U and V, are connected. Kumar et al. hypothesise that the signature of an 
emerging Web community contains at least one "core", which is a complete bipartite 
subgraph with at least i vertices from U and j vertices from V. In their model, the 
vertices in U and V are Web pages and the edges are links from U to V. Seeking an 
core is equivalent to seeking a perfect two-dimensional diamond cube (all cells 
are allocated). 

The iterative pruning algorithm is a specialisation of the basic algorithm we use 
to seek diamonds: it is restricted to two dimensions and is used as a preprocessing 
step to prune data that cannot be included in the (i, j) cores. Although their paper has 
been widely cited II18W231 . it appears that our work is the first to first to propose a 
multidimensional extension to their approach and to provide a formal analysis. 

2.2. Sub-sampling with Database Queries 

Relational Database Management Systems (RDBMS) have optimisation routines 
that are especially tuned to address both basic and more complex SELECT . . . FROM 
. . . WHERE . . . queries. However, there are some classes of queries that are difficult 
to express in SQL, or that execute slowly, because suitable algorithms are not available 
to the underlying query engine. They include skyline, top-A: and nearest-neighbour 
queries. 

2.2.7. Iceberg Queries 

The iceberg query introduced by Fang et al. [4 1 performs an aggregate function over 
a specified dimension list and then eliminates aggregate values below some specified 
threshold. Diamond dicing eliminates slices where the aggregate value of the slice 
falls below a threshold. Unlike icebergs, dimensions can be constrained by different 
thresholds simultaneously. 

2.2.2. Skyline Operator 

The Skyline operator [22 24] seeks a set of points where each point is not "domi- 
nated" by some others: a point is included in the skyline if it is as good or better in all 
dimensions and better in at least one dimension. Attributes, e.g. distance or cost, must 
be ordered. Therefore, indexes can be beneficial to Skyline queries. 
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2.2.3. Top-k 

Another query, closely related to the skyline query, is that of finding the "top-/c" 
data points. For example, we may seek the ten most popular products sold in a store. 
Donjerkovic and Ramakrishnan [25 1 frame the top-Zc problem as one of finding the best 
answers quickly rather than all the answers. Their method is applicable to unsorted and 
unindexed attributes. (Otherwise the index would be used to provide an exact result.) 

Top-A: dominating queries [26 1 combine skyline and top-fc to present a set of points 
that dominate others in all dimensions, but keep the result set to a user-defined size. 

2.2.4. Nearest Neighbours 

One of the most common multidimensional queries is the nearest neighbour query, 
which seeks elements that are "close" to a provided target. For example, given a set of 
users, we might seek users that have a profile similar to the current user. A common 
query asks to find the k nearest neighbours (kNN), that is, k neighbours that are as 
close as possible to the target. 

Reverse nearest neighbours [27 ] start with a given element and asks which possi- 
ble targets would have this element in the nearest neighbours. For example, imagine 
that customers only visit one of the 10 nearest stores. Given a customer, which store 
locations would attract him? 

Nearest neighbour queries are often difficult to index, though some hashing tech- 
niques J28]|29) can help accelerate queries. They also require a specific distance mea- 
sure. 



2.2.5. Outlier Identification 

Another frequent type of query in multidimensional data sets is outlier identifi- 
cation. For example, we might seek elements which are far from most other data 
points 1 30 1 . Points that are not outliers are "near" some fraction of the other data 
sets. That is, we are left with a "core". Meanwhile a diamond cube can be seen as such 
a core. 



2.3. Formal Concept Analysis 

In Formal Concept Analysis j3Tll32"l a Galois (concept) lattice is built from a binary 
relation. It is used in machine learning to identify conceptual structures among data 
sets. A formal concept is a set of objects — extent — together with the set of attributes — 
intent — that describe each of the objects. For example, a set of documents and the set 
of search terms those documents match, form a concept. 

Given the data in Fig. 2a the smallest concept including document 1 is the one with 
documents {1, 2} and search terms {A,B,C,D}. Concepts can be partially ordered by 



inclusion and thus can be represented by a lattice as in Fig. 2b 



Galois lattices are related to diamond cubes: a perfect 3x3 diamond is described 
in the central element of the lattice of Fig. [2b] Formal Concept Analysis is typically 
restricted to two dimensions although Cerf et al. Il33l generalise formal concepts by 
presenting an algorithm that is applied to more than two dimensions. Their definition 
of a closed n— set — a formal concept in more than two dimensions — states that each 
element is related to all others in the set and no other element can be added to this set 
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(a) A 3 X 3 diamond is embedded in this bi- 
nary relation. 



{ 12345, 0} 
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(b) Galois lattice. Each element (concept) in 
the lattice is defined by its extent and intent. 



Figure 2: Documents and search terms in an information retrieval system and the cor- 
responding Galois lattice. 



without breaking the first condition. It is the equivalent of finding a perfect diamond 
in n dimensions. For example, given the data of Table [5] {(a, 7 )(1, 2) (A, B) } is an 
example of a closed 3-set and also a perfect 4,4,4-diamond. 
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Table 2: A 3-dimensional relation with closed 3-set {(a, 7 )(1, 2)(A, B) }. 



2.4. Applications for Diamond Cubes 

Diamonds have the potential to be useful in different areas and we present two 
specific applications as examples. First, we suggest they could be used when generating 
tag clouds — visual indexes that typically describe the content of a Web site. Tag clouds 
are defined and discussed in the next section. Second, one of the original motivations 
for this work was to identify a single large, dense subcube in any given data set. Once 
found, a dense region can be exploited to facilitate visualisation of the data or be used 



for further statistical analysis. Related work in this area is discussed in Section 2.4.2 
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Figure 3: Tags and their corresponding weights. 



2.4.1. Visualisation 

It is difficult to visualise large multidimensional data sets |9] Ell G3 . Aouiche et 
al. proposed tag clouds as an alternative l36l . Tag clouds are used on Web sites to 
communicate the relative importance of facts or topics discussed on a particular page. 
A tag is defined 1 36 1 as a term or phrase describing an object with corresponding non- 
negative weights determining its relative importance, and it comprises the triplet (term, 
object, weight). 

For practical reasons, only the top values in each dimension are presented in a tag 
cloud. They can be selected by a top-fc query on each dimension. However, if we seek 
significant values interacting together in a multidimensional way, then we can use the 
diamond operator. This would ensure that if the user slices on a value, some of the "top" 
values from other dimensions remain relevant. Consider Fig. [3] and its corresponding 
tag cloud. Taking the top-2 from each dimension the cloud comprises goldfish, rabbit, 
Chicago and Montreal. 

If we slice on Chicago using a top-2 query, then goldfish and dog would be part of 
the cloud, which may very well confuse the user as dog was not part of the originally 
presented tags. If however, we slice on Montreal using a top-2 query we have cat and 
rabbit, also potentially confusing. Conversely, a 13 x 13 diamond retains the shaded 
items no matter which slice is chosen. 

2.4.2. Dense Subcubes 

One motivation for the diamond dice is to find a single, non-trivial dense region 
within a data cube. If the region has sufficient density, then hybrid storage becomes 
possible: the dense region can be stored as a multidimensional array, whereas the tuples 
outside of the dense region can be stored sparsely, e.g. as a list of locations. Another 
benefit of partitioning dense and sparse regions is that it enables visual representation 
of data to be enhanced. The dense regions can be represented close together on a single 
screen. Ben Messaoud et al. [9| employ Multiple Correspondence Analysis (MCA) to 
reorganise the cube so that dense subcubes are moved together for easy visualisation: 
A complete disjunctive table representing the entire cube is built and then MCA is 
applied to obtain factorial axes. Eigenvalues for each axis are computed and the user 
chooses a number of axes to visualise. We repeat the example from Ben Messaoud et 
al. in Fig. [4] As with diamond dicing, this approach measures the contribution of a 
set of cells to the overall cube. No timings were given in their paper, so it is difficult 
to compare the efficiency of MCA and diamond dicing, although the outputs are quite 
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(a) Before reorganisation (b) After reorganisation 



Figure 4: The allocated cells are in dark grey: they are moved together by the reorgan- 
isation. 

similar. 

Some researchers seek multiple dense regions rather than a single one, either to 
compute range sum queries more efficiently ll37ll or as a preprocessing step before fur- 
ther statistical analysis l38l . Lee uses a histogram-flattening technique by computing 
three relative density measures for each attribute in each dimension. Dense regions in 
each dimension are then used to build multi-dimensional dense regions. The candidate 
results are then refined iteratively. Experiments conducted on synthetic data seeded 
with clusters show that this method achieves good space reduction. However, there is 
no evidence of experimentation on real data or the time to compute their structure. 

Barbara and Wu fl38l seek to identify dense regions that can be further analysed 
by statistical methods. They shuffle attributes within dimensions based on a similarity 
measure. They do not mention whether rearranging attributes in one dimension has 
an adverse affect on the positioning of attributes in another dimension. Intuitively, 
one would expect there to be some issues with this process. The example provided in 
their paper identified four dense regions and showed how their algorithm would gather 
them together. The diamond dice applied to their example would find the largest dense 
region. If one then applied a diamond dice to the pruned area iteratively, all the regions 
identified by Barbara's method would also be identified by the diamond dice for this 
particular example. 

3. Properties of Diamond Cubes 

In this section a formal model of the diamond cube is presented. We show that 
diamonds are nested, with a smaller diamond existing within a larger diamond. We 
prove a uniqueness property for diamonds and we establish upper and lower bounds on 
the parameter k for both COUNT and SUM-based diamond cubes. 

3.1. Formal Model 

Researchers and developers have yet to agree on a single multidimensional model 
for OLAP [39]. Our simplified formal model incorporates several widely-accepted def- 
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(a) 3-D cube 



(b) Slice on product shoe 



(c) Dice on months March-May 



Figure 5: OLAP terms: cube, slice and dice. 



initions for the terms illustrated in Fig. [5] together with new terms associated specifi- 
cally with diamonds. For clarity, all terms are defined in the following paragraphs. 

A dimension D is a set of attributes that defines one axis of a multidimensional 
data structure. For example, in Fig. [5] the dimensions are location, time and product. 
Each dimension Di has a cardinality rii, the number of distinct attribute values in this 
dimension. Without losing generality, we assume that n\ < n 2 < ... < rid- A 
dimension can be formed from a single attribute of a database relation and the number 
of dimensions is denoted by d. 

A cube is the 2-tuple (D,f) which is the set of dimensions {Di, D 2 , ■ ■ ■ ,Dd} 
together with a total function (/) which maps tuples in D\ x D 2 x • • • x to RU {_L}, 
where _L represents undefined. Fig. 5a shows a cube with three dimensions. 

A cell of cube C is a 2-tuple ((11, x%, ... , Xd) € D\ x D 2 x • ■ ■ x Dd, v) where 
v = f(xi, X2, ■ ■ ■ , Xd) is called a measure. The measure may be a value v € M, in 
which case we say the cell is an allocated cell. Otherwise, the measure is _L and we 
say the cell is empty — an unallocated cell. For the purposes of this paper, a measure 
is a single value. In more general OLAP applications, a cube may map to several 
measures. Also, measures may take values other than real-valued numbers — booleans, 
for example. 

A slice is the cube C = (D' , f) obtained when a single attribute value is fixed 
in one dimension of cube C = (D, /). For example, Fig. 5b is a slice of the cube 
presented in Fig. [5a] 

A dice defines a cube S from an existing cube by removing attribute values and 
the corresponding cells. For example, Fig. 5c illustrates a dice applied to the cube 
from Fig. 5a where all months except March, April and May were removed. The 
resulting cube still has the same number of dimensions. We call it a subcube because 
its dimensions are subsets of the dimensions of the original cube, and, as a function, it 
is a restriction to the corresponding subset of cells. 

An aggregator is a function, a, that assigns a real number to a set of cells — such as 
a slice. For example, SUM is an aggregator: SUM(slicei) = v\ + v 2 + ■ ■ ■ + v m where 
m is the number of allocated cells in slices and the i?j's are the measures. 

A slice 5" is a subset of slice S if every allocated cell in S' is also an allocated cell in 
S. An aggregator a is monotonically non-decreasing if S' C S implies a(S') < cr(S). 
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Figure 6: Part of the Netfiix BOl fact table (cube). Attributes (dimensions) are movie, 
reviewer and date. Each row is a fact (allocated cell). The measure is rating. 

Similarly, a is monotonically non-increasing if S' C S implies cr(S') > cr(S'). Mono- 
tonically non-decreasing operators include COUNT, MAX and SUM over non-negative 
measures. Monotonically non-increasing operators include MIN and SUM over non- 
positive measures. MEAN and MEDIAN are neither monotonically non-increasing, nor 
non-decreasing functions. 

Our formal model maps to the relational model in the following ways: (See Fig. [6]) 

• A cube corresponds to a fact table: a relation whose attributes comprise a pri- 
mary key and a single measure. 

• An allocated cell is a fact, i.e. it is a distinct record in a fact table. 

• A dimension is one of the attributes that compose the primary key. 

3.2. Diamond Cubes are Unique 

Intuitively, a diamond cube is a subcube where all attribute values satisfy a thresh- 
old condition. For example, all selected stores must have total sales over one million 
dollars. We call such conditions carats. 

Definition 3.1. Given a number k, a dimension has k carats if the aggregate of every 
slice along that dimension is at least k. That is, for every slice x, we have cr(x) > k. 

Note that if a dimension has k carats, it necessarily has k! carats for k' g [0, k). 
Given two subcubes A and B of the same starting cube, their union AU B is defined by 
the union of the pairs of dimensions. For example, if A is the result of a dice limiting 
the location to Montreal and B is the result of a dice limiting the location to Toronto, 
the subcube A U B will be the result of a dice limiting the location to both Montreal 
and Toronto. Similarly, the intersection (A n B) is defined by the intersection of the 
pairs of dimensions. We say that subcube A is contained in subcube B if all of the 
dimensions of A are contained in the corresponding dimensions of B. 

For monotonically non-decreasing operators (e.g., COUNT, MAX or SUM over non- 
negative measures), union preserves the carat, as the next proposition shows. 
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(a) Cube with positive and negative measures. 
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Figure 7: There is no unique (4,4) diamond for this cube. 



Proposition 3.1. If the aggregator a is monotonically non-decreasing, then the union 
of any two cubes having fcj ( resp. k^) carats over dimension Di has min(fej , k[) carats 
over dimension Di as well, for i = {1, 2, . . . , d}. 



Proof. The proof follows from the monotonicity of the aggregator. 



□ 



If we limit ourselves to monotonically non-decreasing aggregators, then we can 
efficiently seek the largest possible subcube satisfying a given set of carats. We call 
such a subcube the diamond. 

Definition 3.2. The k\,k%, . . . , kd-carat diamond is the maximal subcube having 
k\, , . . . , kd carats over dimensions D\ , Di . . . , Dd- That is, any subcube having 
ki, 1$2, ■ ■ ■ , kd carats is contained in the diamond. 

By Proposition |3.1| the diamond is unique when a is monotonically non-decreasing: 
it is given by the union of all subcubes having k\,k%, . . .kd carats. For more general 
aggregators, the computation of the diamond might be NP-hard or ill-defined ATI . For 
instance, when SUM is used over cubes having both positive and negative measures, 
there may no longer be a unique solution to the problem 'find the k\,k%. . . /c^-carat 
cube'. This is indeed the case for the cube in Fig. [7] 

Any reference to a /c -carat diamond cube implies k\ = k<i = . . . = kd = k. 

3.3. Diamond Cubes are Nested 

The following proposition show that diamonds are nested. This is helpful because 
the xi,X2, ■ ■ ■ , x^-carat diamond can be derived from the y\ , y^i ■ ■ ■ Ud-camt diamond 
when j/j < Xi for all i. 

Proposition 3.2. The diamond having k' carats over dimensions i\, . . . ,idis contained 
in the diamond having k carats over dimensions i\, . . . ,id whenever k' > k. 

Proof. Let A be the diamond having k carats and B be the diamond having k' carats. 



By Proposition 3.1 AUB has at least k carats, and because A is maximal, AU B = A; 
thus, B is contained in A. □ 
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4. A Priori Bounds on the Carats 



The computations of a diamond requires that the analyst specifies the desired num- 
ber of carats. However, this may not be practical for all dimensions. For example, the 
analyst may want to select stores with sales above one million dollars, but he may not 
know how to select the threshold for the product dimension. In such cases, it might 
be best to set the carats to the largest possible value which generates a non-empty dia- 
mond. Finding this maximal number of carats can be done efficiently by binary search 
if we can determine a limited range of possible values. 

Given a cube C and a, then n is the largest number of carats for which the cube has 
a non-empty diamond. Intuitively, a small cube with many allocated cells should have 
a large k, and the following proposition makes this precise. 

Proposition 4.1. For COUNT-based carats, we have n > |C|/Si( n i — 1) — 3. 

Proof. Solving for k in Corollary |A. l| (see |Appendix A) , we have a lower bound on the 
maximal number of carats: n > \C\/ J2i( n i — 1) — 3. □ 



Based on this lower bound alone, we compute n efficiently (see § 6.4 1. For a related 



discussion on SUM-based diamonds, see Appendix B 



5. Algorithms 

Computing diamonds is challenging because of the interaction between dimen- 
sions; modifications to a measure associated with an attribute value in one dimension 
have a cascading effect through the other dimensions. We use several different ap- 
proaches to compute diamonds: 

• Although there is no publicly available implementation of Kumar et al.'s algo- 
rithm [17], we have developed one that resembles the description provided in 
their paper. We refer to it as Kumar for the remainder of this paper. It has been 
extended to address more than two dimensions. 

• We implemented a custom program in Java that loops through the cube checking 
and updating the COUNT or SUM for all attribute values in each dimension until 
it stabilises (see § sec:external). 

• We also implemented an algorithm using SQL (see § |5.2| i. 

We based both our custom and SQL implementations on the basic algorithm for 
computing diamonds given in Algorithm [T] Its approach is to repeatedly identify an 
attribute value that cannot be in the diamond, and then remove the attribute value and 
its slice. The identification of "bad" attribute values is done conservatively, in that they 
are known already to have a sum less than required (a is SUM), or insufficient allocated 
cells (er is COUNT). When the algorithm terminates, only attribute values that meet the 
condition in every slice remain: a diamond. 

Algorithms based on this approach always terminate, though they might sometimes 
return an empty cube. By specifying how to compute and maintain sums for each 
attribute value in every dimension we obtain different variations. The correctness of 
any such variation is guaranteed by the following result. 
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input: a d— dimensional data cube C, a monotonically non-decreasing 






ag 


greg 


ator a and k\ > 0, k^ > 0, . . . , kd > 


output: the diamond cube A 


stable 


— false 


while ^stable do 




// major iteration 




stable 


— true 




for dim e {1, . . . , d} do 






for i 


in all attribute values of dimension dim do 








Cdim.i f (slice i on dimension dim) 








if C dim ,i < fc dim then 










delete attribute value i 










stable <— false 








end 






end 






end 






end 








return cube without deleted attribute values; 



Algorithm 1: Algorithm to compute the diamond of any given cube by deleting 



slices eagerly. 

Theorem 5.1. Algorithm^is correct, that is, it always returns the k\, &2, • • • , k^-carat 
diamond. 

Proof. Because the diamond is unique, we need only show that the result of the algo- 
rithm, the cube A, is a diamond. If the result is not the empty cube, then dimension Di 
has at least value ki per slice, and hence it has ki carats. We only need to show that 
the result of Algorithm [T] is maximal: there does not exist a larger k\, fc 2 , . . . , A^-carat 
cube. 

Suppose A 1 is such a larger ki, k-z, ■ ■ ■ , fc^-carat cube. Because Algorithm [TJbegins 
with the whole cube C, there must be a first time when one of the attribute values of 
dimension dim of C belonging to A' but not A is deleted. 

At the time of deletion, this attribute's slice cannot have obtained more cells so it 
still has value less than k dlm . Let C be the cube at the instant before the attribute is 
deleted, with all attribute values deleted so far. We see that C is larger than or equal to 
A' and therefore, slices in C corresponding to attribute values of A' along dimension 
dim must have more than fc dim carats. Therefore, we have a contradiction and must 
conclude that A' does not exist and that A is maximal. □ 

5.1. Custom Software 

The size of available memory affects the capacity of in-memory data structures to 
represent data cubes. In our experiments we restricted the size of available memory 
to less than 2 GiB. We are interested in processing large data. Therefore, we seek 
an efficient external memory implementation where the data cube can be stored in an 
external file whilst the important counts (resp. sums) are maintained in memory. 
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cell 


PID 


SID 


month 


i 


1 


a 


Jan 


ii 


1 


b 


Jan 


iii 


2 


c 


Feb 


iv 


2 


d 


Mar 


V 


3 


a 


Feb 


vi 


3 


b 


Feb 


vii 


3 


d 


Apr 


viii 


4 


c 


Apr 


ix 


4 


e 


Jan 



(a) Computing the 2-carat diamond 
would require 3 iterations over this 
cube. 



X X 
X X 

X X 

X X 

X X 

X X 

(b) An nxn cube with 2n allocated cells 
(each indicated by a X ) and a 2-carat dia- 
mond in the upper left: it is a difficult case 
for several algorithms. 



Figure 8: Constructing an SQL query to compute diamonds is challenging 



Algorithm [T] checks the a-value for each attribute on every iteration. Calculating 
this value directly, from a data cube too large to store in main memory, would entail 
many expensive disk accesses. Even with the COUNTS maintained in main memory, 
it is prudent to reduce the number of I/O operations as much as possible. One way 
this can be achieved is to store the data cube as normalised binary integers using bit 
compaction J42l — mapping strings to small integers starting at zero. 

Algorithm [2] employs d arrays, ai to a^, that map attributes to their aggregate a- 
values. As values are pruned from the diamond, we must repeatedly update these 
arrays so that they continue to maintain the aggregate of each slice. This update can be 
executed in constant time for aggregators such as COUNT and SUM: in the notation of 
Algorithm|2] the update is computed as &j(xj) = sy(xj) — cr({r}). 

Each time the algorithm passes through the data, it updates the aggregates eagerly, 
but only marks the cells as deleted. Only when a significant fraction of the cells have 
been marked as deleted are the cells actually deleted: we found that rebuilding the list 
of cells when more than half of the cells are marked as deleted to be efficient (r = 0.5). 
When memory is abundant, we can use Algorithm [2] while keeping the content of the 
files in memory. 



5.2. An SQL-Based Implementation 

A data cube can be represented in a relational database by a fact table. (See Fig. 8a ) 
In this example the dimensions are: productID { 1,2,3,4}, salesPersonID {a,b,c,d,e} and 
month {Jan,Feb,Mar,Apr}. The cell numbers are noted in this example, but they do not 
form part of the cube. 

Formulating a diamond cube query in SQL-92 is challenging since examples can 
be constructed where only one attribute ever has <r(slicedi m .i) < k, but its removal 
exposes another attribute, and so on. See Figure [8b] and consider computing the 2- 



carat diamond from the cube in Fig. 8a Every dimension must be examined three 



times to determine that cells i, ii, v and vi comprise the 2-carat diamond. 
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input: file inFile containing d— dimensional cube C, integers 






ki,k 2 


. . . , kd > 0, monotonically non-decreasing aggregator a, a 






parameter r € [0, 1) 


output: the cells in the diamond data cube 


foreach dimension i do 




Create array a 4 of size D, 




foreach attribute value v in dimension i do 








i(v) 


= <r( slice for value v of dimension i in C) 




end 






end 








stable • 


— false 


while ^stable do 




stable 


— true 




foreach cell r of inFile which is not marked as deleted do 






(0*i. 


x 2 , ■ ■ -,x d ),v) <- r 






Deleted <— false 






for i 


€ {1, ...,d}do 








if &i(xi) < ki then 










// attribute Xi had previously been deleted 










from the diamond 










for j e {1, . . . ,i - l,i + 1, . . . ,d} do 










update a.j (xj) given the removal of cell r 










end 










stable <— false 










mark r as deleted 










break 










// only delete this cell once 








end 






end 






end 








if the fraction of cells marked as deleted exceeds t then 






rebuild inFile without the deleted cells 




end 






end 








rebuild inFile without the deleted cells 


return inFile 



Algorithm 2: Diamond dicing for relationally stored cubes. Each iteration, less 



data is processed. 
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Using nested queries and joins, we could essentially simulate a fixed number of 
iterations of the outer loop in Algorithm[TJ Unfortunately, we cannot bound the number 
of iterations for that loop by a constant, leaving this approach as merely approximating 



the diamond. In Fig. 8b we see an example where the number of iterations I = 
Vt(n) and stopping after o(n) iterations results in a poor approximation with O(n) 
allocated cells and attribute values — whereas the true 2-diamond has 4 attribute values 
and 4 allocated cells. 



INPUT: a d— dimensional data cube C and k > 
OUTPUT: the diamond A 
initialise TZ to C, the fact table 
repeat {major iteration} 

execute the fragment of SQL pseudocode shown below 
until no records were deleted from TZ 
return TZ as A 



CREATE TABLE temp 1 AS 
(SELECT dim! FROM TZ 

GROUP BY dimi HAVING a(measure) < k) ; 



CREATE TABLE temp d AS 
(SELECT dim d FROM TZ 

GROUP BY dim d HAVING a(measure) < k) ; 
DELETE FROM TZ 

WHERE dim x IN (SELECT * FROM temp x ) OR . . . 
dim d IN (SELECT * FROM temp d ) ; 

Algorithm 3: Variation where the inner two loops in Algorithm [1] are computed 
in SQL. This process can be repeated until TZ stabilises. 



We express the essential calculation in SQL, as Algorithm[3] It is implemented as 
a stored procedure, in SQL: 1999, which allows the iterations to be controlled entirely 
within the DBMS. Algorithm [3] can be executed against a copy of the fact table, which 
becomes smaller as the algorithm progresses. The fastest variation of this algorithm 
does not delete slices immediately, but instead updates boolean values to indicate the 
slices are not included in the solution. The data cube is rebuilt when 75% of the re- 
maining cells are marked for deletion. B-tree indexes are built on each dimension to 
facilitate faster execution of the many GROUP BY clauses. 



5.3. Complexity Analysis 

Algorithm [T] visits each dimension in sequence until it stabilises. Ideally, the sta- 
bilisation should occur after as few iterations as possible. 

Let / be the number of iterations through the input file till convergence; i.e. no 
more deletions are done. Value / is data dependent and (by Fig. 8b i is OQ^ n^) in 
the worst case. In practice, / is not expected to be nearly so large, and working with 
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X 
X 

X X 

X X 
X 

Figure 9: The 2-carat diamond requires more iterations to converge than 3-carat dia- 
mond. 

large real data sets / did not exceed 56. Experimentally, it appears that the relationship 
of I to k is bi-tonic, i.e. it is non-decreasing to n + 1 and non-increasing thereafter. 
Unfortunately, there are some cubes for which this is not the case. Fig. [9] illustrates 
such a cube. On the first iteration, processing columns first for the 2-carat diamond, 
a single cell is deleted. On subsequent iterations at most two cells are deleted until 
convergence. However, the 3-carat diamond converges after a single iteration. 

The value of k relative to n does, however, influence /. Typically, when k is far 
from k — either less or greater — fewer iterations are required to converge. However, 
when k exceeds k by a very small amount, say 1, then typically a great many more 
iterations are required to converge to the empty cube. 

Algorithm [2] runs in time 0(Id\C\); each attribute value is deleted at most once. 
Often, the number of attribute values remaining in the diamond decreases substantially 
in the first few iterations and those cubes are processed faster than this bound suggests. 
The more carats we seek, the faster the cube decreases initially. 

6. Experiments 

We show that diamonds can be computed efficiently, i.e. within a few minutes and 
using less than 2 GiB of memory even for very large data sets. Some of the properties 
of diamonds, including their density (count-based diamonds) and the range of values 
the carats may take, were reviewed experimentally. 

6.1. Hardware and Software 

All experiments were conducted on a single Dell PE1950 compute node connected 
to a 16-node cluster through an Infiniband communications link. Each node contains 2 
quad core Intel(R) Xeon(R) E5405 (2.00 GHz) processors with 8 GiB of RAM. 

The algorithms were implemented in Java, using SDK version 1.6.0, and the code 
is archived at a public website [ 16 1. The SQL experiments were conducted on MySQL 
version 5.5 Community Server with MylSAM storage engine and making use of a 
stored procedure. A Java interface connected to the database with mysql-connector- 
java-5.1.15 to collect execution times. Of the index structures available in this version 
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Table 3: Statistics of TWEED, Netfiix, census and weather data. 



cube 


dimensions 


\c\ 




measure 


TW1 


4 


1957 


88 


count 


TW2 


15 


4963 


674 


count 


TW3 


15 


4963 


674 


sum killed 


NF1 


3 


100478 158 


500137 


count 


NF2 


3 


100478 158 


500137 


sum rating 


NF3 


4 


20000000 


473 753 


count 


C1 


27 


135 753 


5 607 


count 


C2 


27 


135 753 


504 


sum stocks 


W1 


11 


124164 371 


48 654 


count 


W2 


11 


124164 371 


48 654 


sum total cloud cover 


Table 4: Statistics of the King James Bible data. 


cube 


dimensions 


\C\ 




measure 


B1 


4 


54601077 


404 


count 


B2 


4 


24000000 


325 


count 


B3 


4 


32000000 


322 


count 


B4 


4 


40000000 


364 


count 


B5 


4 


54601077 


303 


sum 4-gram instances 


B6 


10 


365 231367 


96 


count 



of MySQL, only B-trees are appropriate to the diamond dice operation. Spatial index- 
ing is limited to two dimensions and hash indexing requires that the data reside in main 
memory. 

6.2. Data Used in Experiments 

A varied selection of freely-available real-data sets together with some systemat- 
ically generated synthetic data sets were used in the experiments. Each data set had 
a particular characteristic: a few dimensions or many, dimensions with high or low 
cardinality or a mix of the two, small or large number of cells. They were chosen to il- 
lustrate that diamond dicing is tractable under varied conditions and on many different 
types of data. 

6.2.7. Real Data 

Four of the real-data sets were downloaded from the following sources: 



1. TWEED: http://folk.uib.no/sspje/tweed.htm (43 

2. Netfii x: http://www.netfiixprize.com [44] 

3. Census Income: http://kdd.ics.uci.edu/ B31 



4. Weather: http://cdiac.ornl.gov/ftp/ndp026b/ |46) 
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Table 5: Statistics of the synthetic data cubes. 



Cube 


dimensions 


\c\ 




U1 


3 


999 987 


10773 


U2 


4 


1000000 


14 364 


U3 


10 


1000000 


35 910 


S1 


3 


939 153 


10505 


S2 


4 


999 647 


14296 


S3 


10 


1000000 


35 616 


SS1 


3 


997 737 


74276 


SS2 


4 


999 995 


99 525 


SS3 


10 


1000000 


248 703 



Details of how the cubes were extracted are available at a public website [ 16 1 and their 
statistics are given in Table[3] Each cube was stored relationally in a comma-separated 
file on disk. A brief description of how data cubes were extracted from the King James 
Bible data follows. 

The fifth data set was generated from the King James version of the Bible available 
at Project Gutenberg [47]. KJV-4grams |48 49 1 is a data set motivated by applications 
of data warehousing to literature. It is a large list (with duplicates) of 4-tuples of words 
obtained from the verses in the King James Bible [47], after stemming with the Porter 
algorithm 115011 and removal of stemmed words with three or fewer letters. Occurrence 
of row wi, W2,W3,W4 : indicates a verse contains words wi through Wi, in this order. 
This data is a scaled-up version of word co-occurrence cubes used to study analogies in 
natural language ll5Tll52l . These data were chosen to be representative of large cubes 
that might occur in text mining applications. 

B1 was extracted from KJV-4grams. Duplicate records were removed and a count 
of each unique sequence was kept, which became the measure for cube B5. Four sub- 
cubes of B1 were also processed: B2: first 24 000 000 rows; B3: first 32 000 000 rows; 
and B4: first 40000000 rows. KJV-lOgrams has similar properties to KJV-4grams, 
except that there are 10 words in each row and the process of creating KJV-lOgrams 
was terminated when 500 million records had been generated — at the end of Genesis 
19:30. Cube B6 was extracted from KJV-lOgrams. The statistics for all six cubes are 
given in Table |4] 

6.2.2. Synthetic Data 

To investigate the effect that data distribution might have on the size and shape of 
diamonds, nine cubes of varying dimensionality and distribution were constructed. We 
chose 1 000000 cells with replacement from each of three different distributions: 

• uniform — cubes U1, U2, U3. 

• power law with exponent 3.5 to model the 65-35 skewed distribution — cubes S1 , 
S2, S3. 
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Table 6: Wall-clock times (in seconds) for preprocessing real-world data sets. DB- 
Count and DBSum implement Algorithm [2] for both string and binary data. A ' — ' 
indicates that this algorithm was not applied to the corresponding data cube. 



Cube 


DBCount/DBSum 


Kumar 


SQL 




(Alg.g 




(Alg.g 


B1 


2.9 x 10 2 






B2 


8.4 x 10 1 




1.5 x 10 3 


B3 


1.1 x 10 2 




2.0 x 10 3 


B4 


1.4 x 10 2 




2.6 x 10 3 


C1 


7 x 10° 


1.4 x 10 2 


6 x 10° 


NF1 


2.8 x 10 2 


3.7 x 10 3 


3.6 x 10 3 


NF3 


9.1 x 10 1 


1.0 x 10 3 


1.3 x 10 3 


TW1 


8 x 1(T 2 


1.8 x 10" 1 


1.8 x 10" 1 



• power law with exponent 2.0 to model the 80-20 skewed distribution — cubes 
SS1, SS2, SS3. 

Details of the cubes generated are given in Table [5] 

6.3. Preprocessing Step 

Before applying Algorithm [2] we need to convert the input flat text files to flat 
binary files. To determine if row ordering would have an effect on our binary imple- 
mentation of Algorithm [5J we chose two cubes — C1 and B2 — and shuffled the rows 
using the Unix utility shuf. We compared preprocessing and processing times for each 
of six cubes, averaged over ten runs. Extracting cubes from the data sets included a 
sorting step so that duplicates could be easily removed. We found that preprocessing 
the cube sorted on its dimension of largest cardinality was up to 25% faster than pre- 
processing the shuffled cube. However, execution times for Algorithm [2] were within 
3% for each cube. Therefore, we did not reorder the rows prior to processing. We 
also found no significant difference in execution times when the cubes were sorted by 
different dimensions. 

The algorithms used in our experiments required different preprocessing of the 
cubes. For Algorithm|2]an in-memory data structure was used to maintain counts of the 
attribute values. The Kumar algorithm used d sorted files and Algorithm [3] referenced 
the cube stored in a relational database. Consequently, the preprocessor wrote different 
kinds of data to supplementary files depending on which algorithm was to be used. 

The preprocessing of the cubes was timed separately from diamond building. Pre- 
processed data could be used many times, varying the value for k, without incurring 
additional preparation costs. Table [6] summarises the times needed to preprocess each 
cube in preparation for the algorithms that were run against it. For comparison, sorting 
the Netflix comma-separated data file — using the GNU sort utility — took 18 minutes. 
The Kumar algorithm did not work well with high dimensional cubes. It requires a 
sorted copy of the cube for each dimension which incurs additional space and I/O 
costs. We only ran it against a few cubes. 
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Table 7: Iterations to convergence for sum and count diamonds 



(a) The number of iterations it took to determine the 
K-diamond for COUNT-based diamonds. 



value of k 



cube 


iters 


est. 


actual 


TW1 


6 


19 


38 


NF1 


19 


197 


1004 


NF3 


17 


39 


272 


C1 


8 


282 


672 


W1 


26 


2550 


4554 


B1 


12 


1723 


14 383 


B2 


16 


884 


7 094 


B3 


19 


1098 


8 676 


B4 


12 


1347 


10513 


B6 


5 


57 668 


112232 566 



(b) The number of iterations it took to deter- 
mine the K-diamond on SUM-based diamonds. 
The estimate for re is the tight lower bound from 
Proposition!!? . 1 1 



cube iters 


value of k 
est. actual 


TW3 3 
NF2 40 
C2 5 
W2 19 
B5 4 


85 85 
5 3 483 
1 853 3 600675 
32 20 103 
729 25 632 



6.4. Finding n for COUNT-based Diamonds 



Using Proposition 4. 1 the K-diamond was built for each of the data sets. The initial 



guess (k) for k was the value calculated using Proposition 4. 1 When a non-empty dia- 
mond was built, and for every data set this was indeed the case, k was repeatedly dou- 
bled until an empty cube was returned and a tighter range for n had been established 
Next a simple binary search, which used the newly discovered lower and upper bounds 
as the end points of the search space, was executed. Each time a non-empty diamond 
was returned, it was used as the input to the next iteration of the search. When the 
guess overshot k and an empty diamond was returned, the most recent non-empty cube 
was used as the input. 

Statistics are provided in Table [7a] The estimate of k comes from Proposition |4T] 
and the number of iterations recorded is the number needed to find the K-diamond. The 
estimates for k varied between 4% and 50% of the actual value and there is no clear 
pattern to indicate why this might be. Two very different cubes both had estimates 
that were 50% of the actual value: TW1 , a small cube of less than 2 000 cells and low 
dimensionality, and W1 , a large cube of 123 x 10 6 cells with moderate dimensionality. 

6.5. Finding nfor SUM-based Diamonds 

From Proposition B.2 we have that minj(maxj(cr(slice : , (Di)))) is an upper bound 



for k of any sum-diamond and from Proposition |B.1| a lower bound is the maximum 
value stored in any cell. Indeed, for cube TW3 the lower bound is the k value. For this 



2 This method was more efficient than using the theoretical upper bound on re established in Proposi- 
tion |A.l| 
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Figure 10: Times and iterations needed to generate diamonds with different fc-values 
(labelled). In each case more time and iterations are required for fc-values that slightly 
exceed k. 



reason, the approach to finding k for the sum-diamonds varies slightly in that the first 
guess for fc should be the lower bound + 1 . If this returns a non-empty diamond, then a 
binary search over the range [lower bound +l-upper bound] is used to find k. Statistics 



are given in Table 7b 



6.6. Comparison of Algorithm Speeds 

Neither the initial file size nor the number of cells pruned from each fc-diamond 
alone explains the time necessary to generate each diamond. 

In an earlier implementation of the diamond dicing algorithm BTll . we had observed 
that the time expended was proportional to the number of cells processed. This is not 
as evident in the current implementation, where a new file is written when 50% of the 
cells are marked for deletion, instead of at every iteration. 

Table|8] compares the speeds of Algorithm|3](SQL4) and the binary version of Al- 
gorithm |2](BIN) run against some of the data sets. Times were averaged over five 
runs and then normalised against the binary version. Two versions of BIN were im- 
plemented: each maps the data cube into memory. Whenever the data cube is small 
enough (< 500 Mb) we use our in-memory approach. For the larger cubes, an external- 
memory approach, which maps chunks of the cube into memory, is used. We see that 
our binary implementation effects greater speed-up as the cube size increases and the 
cube density decreases. For example, we see that BIN is 146 times faster than SQL4 
on cube B2, but 498 times faster on the more sparse cube, NF3. 

6. 7. Diamond Size and Dimensionality 

The size (in cells) of the K-diamond of the high-dimensional cubes is large, e.g. the 
^-diamond for B6 captures 30% of the data. How can we explain this? Is this property 
a function of the number of dimensions? To answer this question the K-diamond was 
generated for each of the synthetic cubes. Estimated k, its real value and the size in cells 
for each cube are given in Table [9] The K-diamond captures 99% of the data in cubes 
U1, U2 and U3 — dimensionality has no effect on diamond size for these uniformly 
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Table 8: Relative slowdown of the SQL algorithm compared to the binary implemen- 
tation. Times were averaged over five runs and then normalised against the binary 
execution times. SQL processing for NF1 was forcibly terminated after 19 hours 





Cube 


BIN (s) 


SQL4 (s) 


SQL4 / BIN 






TW1 


0.01 




0.04 


4 






C1 


0.37 




23.0 


62 






B2 


36 




5 291 


146 






B3 


49 




7989 


163 






B4 


57 




10553 


183 






NF3 


10 




4988 


498 






NF1 


124 




<g> 







Table 


9: High dimensionality does not affect diamond size. 


Cube dimensions 


iters 


value of k 


size (cells) 


% captured 








est. 


actual 






U1 


3 


6 


89 


236 


982618 


98 


U2 


4 


6 


66 


234 


975 163 


98 


U3 


10 


7 


25 


229 


977 173 


98 


S1 


3 


9 


90 


1141 


227 527 


24 


S2 


4 


14 


67 


803 


231737 


23 


S3 


10 


14 


25 


208 


260 864 


26 


SS1 


3 


18 


11 


319 


122 878 


12 


SS2 


4 


19 


7 


175 


127 960 


13 


SS3 


10 


17 


1 


28 


165 586 


17 



distributed data sets. Likewise, dimensionality did not affect the size of the ^-diamond 
for the skewed data cubes as it captured between 23% and 26% of the data in cubes 
S1 , S2 and S3 and between 12% and 16.5% in the other cubes. These results indicate 
that the dimensionality of the cube does not affect how much of the data is captured by 
the diamond dice. 



6.8. Iterations to Convergence 



In Section 5.3 we observed that in the worst case it could take n i) iterations 

before the diamond cube stabilised. In practice this was not the case. (See Table [Taj 
) All cubes converged to the K-diamond in less than 1% of J^. n, , 



Table 



7b 



and Fig. 
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with the exception of the small cube TW1 , which took less than 7% ^ n*. Algorithm[2] 
required 19 iterations and 2 minutes^] to compute the 1 004-carat ^-diamond for NF1 
and it took 50 iterations and an average of 3 minutes to determine that there was no 



averaged over 10 runs 
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Figure 11: Cells remaining after each iteration of Algorithm|2]for k = 1004, 1005 and 
1006 on cube NF1 



1 005-carat diamond. We measured the number of cells remaining in cube NF1 after 
each iteration of different fc-diamonds to determine how quickly the diamond converges 
to an empty diamond when k exceeds k. Fig. 1 1 shows the number of cells present in 
the diamond after each iteration for 1 004-1 006 carats. The curve for 1 006 reaches 
zero first, followed by that for 1 005. Since k(NF1 ) = 1 004, that curve stabilises at a 
nonzero value. It takes longer to reach a critical point when k only slightly exceeds k. 

Intuitively, one should not be surprised that more iterations are required when k w 
k: attribute values that are almost in the diamond are especially sensitive to other 
attribute values that are also almost in the diamond. 

The number of iterations required until convergence for all our synthetic cubes 
was also far smaller than the upper bound, e.g. cube S3: 35 616 (upper bound) and 
14 (actual). We had expected to see the uniformly distributed data taking longer to 
converge than the skewed data. This was not the case: in fact the opposite behaviour 
was observed. (See Table [9]) For cubes U1, U2 and U3 the diamond captured 98% 
of the cube: less than 23 000 cells were removed, suggesting that they started with a 
structure very like a diamond but for the skewed data cubes — S1 , S2, S3, SS1 , SS2 
and SS3 — the diamond was more "hidden". 



7. Conclusion 

We presented a formal analysis of the diamond cube. We have shown that, for 
the parameter k associated with each dimension in every data cube, there is only one 
ki,k 2 , ■ ■ ■ fcrf-diamond. By varying the fcj's we get a collection of diamonds for a cube. 
We established upper and lower bounds on the parameter k for both COUNT and SUM- 
based diamond cubes. 
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We have designed, implemented and tested algorithms to compute diamonds on 
real and synthetic data sets. Experimentally, the algorithms bear out our theoretical 
results. An unexpected experimental result is that the number of iterations required 
to process the diamonds with k slightly greater than n is often twice that required to 
process the K-diamond, which also resulted in an increase in running time. 

We have shown that computing diamonds for large data sets is feasible. Our binary 
implementation fared better on large, sparse data cubes than other approaches and these 
results confirm that this algorithm is scalable. 

7.1. Future Research Directions 

Although it is faster to compute a diamond cube using our implementation than 
using the standard relational DBMS operations or OLAP operators, the speed does not 
conform to the OLAP goal of near constant time query execution. Different approaches 
could be taken to improve execution speed: compress the data so that more of the cube 
can be retained in memory; use multiple processors in parallel; or, if an approximate 
solution is sufficient, we might process only a sample of the data. These are some of 
the ideas to be explored in future work. 

Data cubes are often organised with hierarchies of relationships within dimensions. 
For example, a time dimension may include aggregations for year, month and day. Our 
current work does not address the issue of hierarchies and how they might be exploited 
in the computation of diamonds. This is also a potential avenue for future work. 
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Appendix A. Properties of COUNT-based Diamonds 



Proposition A.l. Given the sizes of dimensions of a COUNT-based diamond cube, rii, 
an upper bound for the number of carats k of a non-empty subcube is Jlf=i n i- An 
upper bound on the number of carats hi for dimension i is Oj=i j^i n i- 

Proof. Consider a perfect diamond, i.e. all cells are allocated. We can extract a slice 
from this diamond by holding a value in some dimension, D x , constant. The number 
of allocated cells in this slice will be the product of cardinalities of the remaining di- 
mensions. If we choose D x to be the dimension of largest cardinality, then the number 
of allocated cells in this slice is the carat (k) count for this diamond. Since all cells 
are allocated, there is no non-empty diamond with more than k carats. The rest of the 
result follows similarly. □ 

An alternate upper bound on the number of carats in any dimension is |C|, the 
number of allocated cells in the cube. For sparse cubes, this bound may be more useful 



than that from Proposition A. 1 



Intuitively, a cube with many carats needs to have many allocated cells: accord- 
ingly, the next proposition provides a lower bound on the size of the cube given the 
number of carats. 

Proposition A.2. For d > 1 and a = COUNT, the size \C\ of a d-dimensional non- 
empty cube Cofk carats satisfies \ C\ > krid > k d / more generally, a non-empty 
k\, J$2 , ■ ■ ■ , kd-carat cube has size satisfying 

\C\>maxk ini > (Q*i) 1/(d-1) .ie {1,2,. 

Proof. Pick dimension D^: the cube has rid slices along this dimension, each with k 
allocated cells, proving that \C\ > krid- 

We have that krid > k(J2i n>i)/d — since the average number of allocated cells per 
slice must be less than or equal to max ie {i ,2,...,d} n i — so that \C\ is at least k(^2 i rii)/d. 
If we prove that £\ n i > dfc 1/(d_1) then we will have that n^/d > k d / (d ~ 1 \ 

This result can be shown using Lagrange multipliers. Consider the problem of min- 
imising J^i n t 8 iven the constraints rii=i,2,...,j-i,i+i,...,d rii > k for j = 1,2, ... , d. 
These constraints are necessary since all slices must contain at least k cells. The cor- 
responding Lagrangian is L = J2i ™i + Ej V'll, 1.2 ; • : ,/"< ~ k )- B Y 

inspection, the derivatives of L with respect to m,n2,.. ■ ,nd are zero and all con- 
straints are satisfied when rii = n 2 = . . . = rid = fc 1 '^ -1 ). For these values, 
E. rii = dfc 1 /^ -1 ) and this must be a minimum, proving the result. The more gen- 
eral result follows similarly, by proving that the minimum of n i^i i s reached when 
ni = (n«=i a hY'^-^/ki for all i's. ' " □ 

We calculate the volume of a cube C as Y\a=i n i anc ^ ' ts density is the ratio of 
allocated cells, |C|, to the volume (1^1 / Oi=i ^i)- We can exploit the relationship 
between the size of a diamond and its density to determine whether a diamond of 
particular carat exists in a given data cube. 
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Theorem A.l. For COUNT-based carats, if a cube does not contain a non-empty It- 
carat subcube, then it has at most 1 + (k — 1) X^=i( n « ~ -0 allocated cells. Hence, 
it has density at most (1 + (fc — 1) 53j=i ( n i ~~ -0)/ Ili=i n i- More generally, a cube 
that does not contain a non-empty k\ , k%, . . . , k^-carat subcube has size at most 1 + 
Z)i=i(ki ~ l )( n i - 1) and density at most (1 + ^ i=1 (fc; _ _ 1 ))/Ili=i 

Proof. Suppose that a cube of dimension at most ni x n 2 x • • ■ x contains no k- 
carat diamond. Then one slice must contain at most k — 1 allocated cells. Remove this 
slice. The amputated cube must not contain a fc-carat diamond. Hence, it has one slice 
containing at most k — 1 allocated cells. Remove it. This iterative process can continue 
at most J2i( n i ~~ 1) times before there is at most one allocated cell left: hence, there 
are at most (fc — 1) — 1) + 1 allocated cells in total. The more general result 

follows similarly. □ 



Corollary A.l follows from Theorem|A.l 



Corollary A.l. A cube of size greater than 1 + (k — 1) i n i ~ 1) allocated cells, 
that is, having density greater than 

i + (fc-i)Eti 

nd ' 

must contain a non-empty k-carat subcube. If a cube contains more than 1+X)f=i(^ — 
l)(rii — 1) allocated cells, it must contain a non-empty k\, &2j • • • > k^-carat subcube. 



Proof. From Theorem A. 1 we have that a cube cannot have more than 1 + Yli=i (^i 



— 1) allocated cells unless it also contains a non-empty fc-diamond. Therefore, 
if a cube has more than 1 + J2i=i(^i ~ — 1) allocated cells, it must contain a 

nonempty fc-diamond. □ 



Appendix B. Bounding the carats for SUM-based Diamonds 

For SUM-based diamonds, the goal is to capture a large fraction of the sum. The 
statistic, k, of a SUM-based diamond is the largest sum for which there exists a non- 
empty diamond: every slice in every dimension has sum at least k (see §[4j. Proposi- 



tions B.l and B.2 give tight lower and upper bounds respectively for k. 

Proposition B.l. Given a non-empty cube C and the aggregator SUM, a tight lower 
bound on k is the value of the maximum cell (m). 

Proof. The K-diamond, by definition, is non-empty, so it follows that when the k- 
diamond comprises a single cell, then k takes the value of the maximum cell in C. 
When the K-diamond contains more than a single cell, m is still a lower bound: either 
k is greater than or equal to m. □ 

Given only the size of a SUM-based diamond cube (in cells), there is no upper 
bound on its number of SUM-carats. However, given its sum, say S, then it cannot have 
more than S SUM-carats. We can determine a tight upper bound on k as the following 
proposition shows. 
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Proposition B.2. A tight upper bound for k is 

min(max(sUM(slice,(£)j)))) for i £ {1, 2, . . . , d} and j G {1,2,..., rij}. 

* j 

Proof. Let X = {slice., (A) | SUM(slicej(A)) = max fe (sUM(slice fe (L' i ))} then there 
is one slice x whose SUM(x) is smaller than or equal to all other slices in X. Suppose k 
is greater than SUM(x) then it follows that all slices in this K-diamond must have SUM 
greater than SUM(i). However, x is taken from X, where each member is the slice for 
which its SUM is maximum in its respective dimension, thereby creating a contradic- 
tion. Such a diamond cannot exist. Therefore, min^maxj (sUM(slice., (-D,)))) is an 
upper bound for n. 

To show that min i (max J (sUM(slice :; (Di)))) is also a tight upper bound we only 
need to consider a perfect cube where all measures are identical. □ 

We can also provide a lower bound on the sum of a fc-carat diamond as the next 
lemma indicates. 

Lemma B.l. If the diamond of size s\ x s 2 x • • • x s d has k-carats, then its sum is at 
least k max(si) i € {1, 2, ... , d}. 

Proof. We know that the sum is at least as large as the sum of max(sj), and each is 
at least as large as k, from which the result follows. □ 
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